How to Claim a Discovery

Setting limits for new particles or decay modes has been an active research area for many years. In high energy physics it received renewed interest with the unified method by Feldman and Cousins [1]. Giunti [2] and Roe and Woodroofe [3] gave variations of the unified method, trying to resolve an apparent anomaly when there are fewer events in the signal region than expected. They all discuss the problem of setting limits for the case of a known background rate. The case of an unknown background rate was discussed in a conference talk by Feldman [4] and a method for handling this case was developed by Rolke and López [5]. Little work has been done though on the question of claiming a discovery. This problem could be handled by finding a confidence interval and claiming a discovery if the lower limit is positive. Instead the question of a discovery should be done separately, by performing a hypothesis test with the null hypothesis Ho:”There is no signal present”. Rejecting this hypothesis will then lead to a claim for a new discovery. In carrying out a hypothesis test one needs to decide on the type I error probability α, the probability of falsely rejecting the null hypothesis. This is of course equivalent to the major mistake to be guarded against, namely that of falsely claiming a discovery. In practice a hypothesis test is often carried out by finding the p-value. This is the probability that an identical experiment will yield a result as extreme (with respect to the null hypothesis) or even more so given that the null hypothesis is true. Then if p < α we reject H0; otherwise we fail to do so. For the test discussed here it is not possible to compute the p-value analytically, and therefore we will find the p-value via Monte Carlo. Maybe the most important decision in carrying out a hypothesis test is the choice of α, or what we might call the discovery threshold. As we shall see, this decision is made much easier by the method described here because we will need only one threshold, regardless of how the analysis was done. What a proper discovery threshold should be in high energy physics is a question outside the scope of this paper, although we might suggest α = 0.001 (roughly equivalent to 3σ). Sinervo [6] argues for a much stricter standard of 5σ, or α = 2.9∗10. We believe that such extreme values were used in the past because it was felt that the calculated p values were biased downward by the analysis process, and a small α was needed in order to compensate for any unwittingly introduced biases. If we were to trust that our p-value is in fact correct, a 1 in 1000 error rate should to be acceptable. A general introduction to hypothesis testing with applications to high energy physics is given in Sinervo [6]. A classic reference for the theory of hypothesis testing is Lehmann [7].